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The  image  above  shows  a  Coupled  Ocean/ Atmosphere  Mesoscale  Prediction  System  (COAMPS) 
48-hour  simulation  of  hurricane  Katrina.  The  three-way  coupling  between  CO  AMPS,  the  Navy 
Coastal  Ocean  Model  (NCOM),  and  the  Simulating  WAves  Nearshore  (SWAN)  model  is 
achieved  through  ESMF.  This  simulation  uses  three  atmospheric  domains,  two  ocean  domains, 
and  one  wave  domain.  The  innermost  atmospheric  nest  automatically  translates  along  with  the 
storm.  COAMPS  predicts  a  cold  sea  surface  temperature  anomaly  (degrees  C,  color  shaded  area) 
to  the  right  of  the  hurricane  track  in  response  to  the  passage  of  the  hurricane’s  high  wind  speeds 
(m/s,  black  contours).  The  significant  wave  height  (m,  white  contours)  shows  increased  wave 
growth  on  the  northern  quadrant  of  the  hurricane  due  to  wind-wave-current  coupling  effects.  The 
white  squares  denote  the  observed  location  of  the  eye  at  six  hourly  intervals  and  the  dotted  line 
denotes  the  model  location  of  the  eye  at  hourly  intervals  beginning  at  1 200  UTC  27  August. 
Image  courtesy  of  Sue  Chen ,  Naval  Research  Laboratory . 
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Executive  Summary 

The  Earth  System  Modeling  Framework  (ESMF)  is  in  its  seventh  year,  and  its  component 
interfaces  have  been  implemented  in  most  of  the  prominent  climate  and  weather  codes  in 
the  U.S.:  CCSM,  WRF,  NASA  GEOS-5,  EMC  GFS/NEMS,  NRL  COAMPS  and  the 
GFDL  MOM4  ocean  (see  Appendix  A  for  a  table  of  acronyms).  In  general, 
componentization  with  ESMF  has  been  implemented  at  the  level  of  major  physical 
domains,  where  simulated  interactions  require  inter-component  data  communications 
(e.g.  atmosphere,  ocean).  This  implementation  is  compatible  with  codes  such  as  WRF 
that  impose  their  own  interoperability  conventions  at  the  level  of  individual  physics 
parameterizations. 

The  codes  listed  above,  plus  the  full  GFDL  modeling  system  under  FMS,  can  now 
operate  within  a  component-based,  hierarchical  architecture  in  which  large-scale 
components  can  be  called  as  subroutines.  This  was  not  the  case  when  ESMF  began.  The 
current  level  of  ESMF  adoption  represents  a  major  and  necessary  step  towards  enabling 
U.S.  centers  to  exchange  large-scale  components,  and  is  an  excellent  basis  for  further 
advances  in  code  interoperability. 

The  technical  focus  of  2009  was  on  continued  development  of  ESMF  grid  classes  and  a 
regridding  capability.  These  are  built  on  a  base  of  two  flexible  data  representations  that 
were  constructed  in  previous  years:  an  unstructured  finite  element  mesh  library  and  a 
general  array  class  designed  to  support  fields  on  logically  rectangular  grids.  At  this  point 
ESMF  can  produce  non-conservative  interpolation  weights,  in  parallel  or  serial,  2D  or 
3D,  for  grids  that  can  be  represented  by  triangles  or  quadrilaterals.  Users  can  also  read  in 
external  interpolation  weights,  from  a  package  such  as  SCRIP,  if  they  need  an 
interpolation  option  ESMF  does  not  provide. 

The  organizational  focus  of  2009  was  on  moving  the  ESMF  core  development  team  and 
members  of  the  ESMF  sister  project  Earth  System  Curator  from  NCAR  to  NOAA 
ESRL/CIRES.  These  staff  members  now  comprise  the  NOAA  Environmental  Software 
Infrastructure  and  Interoperability  (NESII)  group.  This  involved  transferring  ten  project 
members  along  with  infrastructure  such  as  websites  and  mailing  lists.  The  strategic 
motivation  for  the  move  was  better  positioning  of  the  development  team  with  respect  to 
its  growing  base  of  operational  and  other  federal  center  users. 

On  the  applications  side,  there  were  several  main  activity  areas  in  2009:  completing 
unstructured  mesh  methods  and  C  interfaces  for  a  BEI  coupling  project,  starting  to 
address  issues  related  to  running  ensembles  with  the  NASA  GEOS-5  model  and  NEMS, 
and  completing  the  implementation  of  ESMF  in  CCSM  in  time  for  the  first  releases  of 
CCSM4  in  early  2010. 

ESMF  management  participated  in  the  definition  of  two  programs:  the  National  Unified 
Operational  Prediction  Capability  (NUOPC),  a  collaboration  that  is  developing  a  multi¬ 
model  ensemble  for  operational  NWP;  and  the  NOAA  Global  Interoperability  Program 
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(GIP),  a  broad  collaboration  that  addresses  software  infrastructure  across  a  set  of  high- 
priority  activities  including  CMIP5/IPCC  AR5.  ESMF  is  an  integral  part  of  both 
programs  and  funded  work  will  begin  in  2010:  for  NUOPC,  development  of  component 
and  coupler  templates;  and  for  GIP,  more  advanced  use  of  ESMF  in  CCSM  and 
assimilation  of  the  NASA  MAPL  usability  layer  into  the  main  ESMF  distribution.  Under 
a  new  TeraGrid  project  and  GIP,  the  ESMF  team  will  be  working  with  NCAR  and 
Purdue  University  on  a  friendly  web-based  interface  and  workflow  system  for  CCSM  and 
potentially  other  climate  models,  based  on  a  Purdue  CCSM  Portal  that  is  currently  used 
in  their  university  courses.  A  recently  awarded  three  year  grant  from  NSF  will  also 
enable  the  Curator  and  ESMF  teams  to  partner  with  GFDL,  ESG,  University  of  Michigan 
and  the  CSDMS  hydrology  consortium  to  develop  on-line  governance  functions  for 
community  modeling  projects. 

Major  accomplishments  during  2009  included: 

•  Delivery  of  ESMF  v4.0.0r,  in  October  2009.  This  release  contained  a  close  to 
complete  implementation  of  the  Grid  class  and  regridding  capability. 

•  Inclusion  of  ESMF  interfaces  for  major  components  in  the  alpha  release  of  the 
Community  Climate  System  Model  4  (CCSM4)  in  January  2010.  CCSM4  also 
uses  the  ESMF  off-line,  higher  order  regridding  package  for  its  standard  non¬ 
conservative  remapping. 

•  Successful  movement  of  the  ESMF  core  team  from  NCAR  to  NOAA  in 
November  2009. 

•  Initial  agreement  on  organizational  changes,  reflected  in  a  new  draft  of  the  ESMF 
Project  Plan,  that  enable  ESMF  to  operate  at  a  federal  center,  and  that  reflect  its 
role  in  NUOPC. 

With  respect  to  the  ESMF  strategic  goal  of  furthering  the  integration  of  models  and  data 
portals,  the  Curator  project  delivered  a  metadata  display  capability  for  CMIP5/IPCC  AR5 
that  will  be  included  in  the  next  public  ESG  Gateway  release.  This  links  the  CMIP5 
datasets  to  comprehensive  descriptions  of  the  models  that  generated  them.  This  work 
was  done  in  collaboration  with  ESG,  GFDL,  PCMDI,  a  host  of  European  centers,  and 
other  collaborators  under  the  auspices  of  the  GO-ESSP  consortium. 
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1  Introduction 


The  Earth  System  Modeling  Framework  (ESMF)  was  motivated  by  the  growing 
complexity  of  building  and  coupling  Earth  system  models.  ESMF  provides  a  set  of 
standard  software  interfaces  and  high-performance  tools  for  common  modeling  functions, 
thereby  promoting  interoperable  software  systems  and  code  reuse.  It  has  transitioned 
from  its  initial  NASA  sponsorship  to  multi-agency  support  and  management.  It  is  the 
technical  basis  for  the  DoD  Battlespace  Environments  Institute,  the  NUOPC,  the  NASA 
Modeling  Analysis  and  Prediction  Program  (MAP),  and  a  host  of  smaller  programs  and 
projects.  One  of  its  offshoots  is  the  Earth  System  Curator  project,  which  has  focused  on 
creating  end-to-end  workflows  involving  high  performance  models  and  data  distribution 
portals.  More  information  about  ESMF  and  its  applications  is  available  on  the  ESMF 
website,  http://www.earthsvstemmodeling.org:  Curator  is  at 
http://www.earthsvstemcurator.org.  and  a  summary  of  projects  is  provided  in 
Appendix  C. 


2  Software  framework  description 

ESMF  components  are  arranged  in  a  hierarchical  structure  to  form  modeling  applications. 

The  current  ESMF  distribution  contains: 

•  Tools  for  building  scientific  components  and  couplers,  and  a  set  of  utilities  for 
common  modeling  functions  (e.g.,  calendar  management,  data  communications) 

•  Scalability  of  core  communication  routines  to  tens  of  thousands  of  processors 

•  Concurrent  or  sequential  execution,  single  or  multiple  executable  modes 

•  Support  for  configuring  ensemble  members  sequentially  or  concurrently 

•  Fortran  interfaces  and  complete  documentation,  select  C  interfaces 

•  Support  for  30+  platform/compiler  combinations,  including  IBM,  Cray  XT,  SGI, 
Linux,  Mac,  Windows,  and  other  platforms.  The  complete  list  is  available  on  the 
ESMF  website.1 2 

3  Timeline  and  strategic  objectives 

ESMF  strategic  objectives  are  described  in  detail  in  the  ESMF  Strategic  Plan  for  2008- 

20 101.  As  outlined  in  the  Plan,  there  are  currently  three  main  activity  areas: 

1 .  Completing  the  functional  requirements  of  the  framework  and  improving  its  ease 
of  use 

2.  Developing  capabilities  and  strategic  partnerships  that  help  to  integrate  the 
framework  into  end-to-end  computational  environments  (science  gateways) 

3.  Supporting  users  who  are  integrating  ESMF  into  applications. 


1  http://www.earthsvstemmodelinq  ora/download/platforms/ 

2  http://www.esmf.ucar.edu/plans/plan  0802  esmfstrat.doc 
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ESMF’s  second  funding  cycle,  under  NOAA,  NASA,  DoD,  and  NSF  sponsors,  will  be 
completed  at  the  end  of  201 0.  Over  the  next  year,  the  ESMF  group  anticipates  one  more 
major  release  increment:  ESMF  5,  which  will  focus  on  standardization  of  interfaces  and 
ease  of  use.  ESMF  interfaces  after  version  5  will  be  backwards-compatible  so  that 
updating  to  new  versions  should  not  require  user  code  changes.  Other  deliverables  in  this 
time  frame  will  include  new  and  enhanced  science  gateways  that  build  on  ESMF 
concepts,  capabilities  and  components. 

The  ESMF  team  will  begin  work  in  201 0  under  two  programs,  GIP  and  NUOPC,  which 
will  help  to  shape  future  strategic  objectives.  The  ESMF  organization  will  experience 
some  organizational  changes  due  to  its  relationship  with  these  programs  and  to  the  move 
of  the  core  development  team  to  NOAA  in  late  2009.  An  updated  version  of  the  ESMF 
Strategic  Plan  covering  2010-201 5  will  be  prepared  this  year. 

All  ESMF  activities  remain  rooted  in  the  project’s  core  values  of  community  ownership, 
distributed  development,  and  open  access  to  information. 

4  Accomplishments  in  brief  for  FY2009 

Framework  development.  During  2009,  the  ESMF  team  was  able  to  build  rapidly  on 
mesh  and  array  data  structures  that  had  been  redesigned  for  greater  flexibility  during  the 
previous  two  years.  One  of  the  highlights  of  a  new  public  release,  ESMF  4.0.0r,  is  a 
remapping  utility  that  produces  interpolation  weights,  in  parallel  or  serial,  2D  or  3D,  for 
grids  that  can  be  represented  by  triangles  or  quadrilaterals.  The  remapping  utility 
supports  bilinear  or  higher  order  weights.  A  conservative  algorithm  is  being  developed 
for  release  in  2010  in  collaboration  with  mathematicians  from  NCAR’s  IMaGe  group. 

Science  Gateways.  The  Curator  teams  partnered  with  the  DOE- funded  Earth  System 
Grid  (ESG)  project,  GFDL,  the  E.U.  METAFOR  project  and  others  on  development  of  a 
metadata  display  for  CMIP5/IPCC  AR5.  This  links  the  datasets  associated  with  CMIP5 
with  comprehensive  scientific  and  technical  descriptions  of  the  models  used  to  generate 
them.  This  is  an  important  improvement  over  AR4,  for  which  relatively  little  on-line 
information  was  available.  The  display  will  also  be  applicable  to  other  model 
intercomparison  projects,  and  builds  on  a  prototype  gateway  constructed  for  a  2008 
NCAR  ASP  colloquium  in  which  a  number  of  atmospheric  dynamical  cores  were 
compared.  The  display  has  been  included  in  initial  releases  of  a  new  version  of  ESG  and 
will  be  part  of  its  next  major  public  release,  currently  scheduled  for  delivery  in  March 
2010. 

Student  visitors  contributed  prototypes  for  the  interaction  of  ESMF  codes  with  data 
portals.  One  showed  that  a  code  instrumented  with  ESMF  could  produce  time  series 
metadata  that  conformed  to  the  requirements  for  a  hydrology  data  portal.  One 
collaborated  with  NRL  to  produce  a  version  of  COAMPS  that  was  instrumented  with 
model  metadata.  Another  instrumented  CCSM  with  metadata,  then  embedded  that 
version  in  an  automated  workflow  that  configured  the  model,  ran  it,  and  documented  the 
run.  These  projects  are  building  towards  an  increased  level  of  automation  and  semantic 
interoperability  that  will  be  an  important  element  of  the  next  ESMF  development  cycle. 
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ESMF  applications.  The  current  estimate  is  that  the  number  of  available  science 
components  has  grown  from  69  at  end  FY2008  to  90  at  end  FY2008.  A  full  listing  of 
components  is  available  on  the  ESMF  website.3 

5  Plans  in  brief  for  FY2010 

Framework  development.  The  focus  for  2010  will  be  on  completing  the  scope  of  work 
under  current  awards,  preparing  ESMF  for  deployment  at  operational  centers  throughout 
the  U.S.  as  part  of  the  NUOPC  program,  and  planning  ahead  for  the  broader  goals  of  the 
GIP  program. 

The  current  ESMF  Strategic  Plan  describes  a  number  of  development  tasks  that  remain  to 
be  completed  before  the  end  of  2010: 

•  Completing  multi-tile  grids  and  exchange  grids 

•  Completing  a  conservative  regridding  capability 

•  Completing  a  basic  but  usable  I/O  system 

•  Completing  a  component  compliance  checker 

In  addition,  during  2010  the  development  team  will  focus  on  standardization  of  interfaces 
and  behavior  across  the  framework,  documentation,  and  the  development  of  training 
materials. 

The  ESMF  team  will  also  begin  work  on  new  NUOPC,  GIP,  and  TeraGrid  projects. 
Development  of  the  NUOPC  usability  layer  will  commence  with  the  construction  of 
component  and  coupler  templates.  Projects  under  GIP  include  integrating  MAPL  into  the 
main  ESMF  source  distribution,  so  that  these  capabilities  are  supported  by  the  ESMF 
team  rather  than  NASA,  and  continued  integration  of  ESMF  into  CCSM.  New  CCSM 
work  includes  exploration  of  the  ways  that  ESMF  regridding  can  be  used  within  the 
model. 

Science  Gateways.  The  Curator  collaboration  will  continue  development  of  the  metadata 
display  for  CMIP5/AR5,  addressing  any  issues  that  arise  after  it  is  fully  deployed. 

Under  both  TeraGrid  and  GIP  funding,  Curator,  Purdue  and  ESG  collaborators  will 
develop  a  GUI  and  workflow  system  for  climate  modeling  based  on  the  Purdue  CCSM 
Portal.  This  Portal,  already  used  in  classes  at  Purdue,  enables  CCSM  to  be  configured 
easily,  and  future  work  will  allow  metadata  and  model  output  data  to  be  archived  back  to 
ESG. 

The  Curator  team  will  also  begin  work  on  a  new  NSF  award  to  develop  the  idea  of 
“commodity  governance,”  which  involves  embedding  group  workspaces  and 
administrative  and  management  functions  into  gateways  to  that  distributed  teams 
involved  in  community  modeling  can  self-organize  more  effectively. 


3  http://www.esmf. ucar.edu/components/ 
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ESMF  applications.  In  2010,  a  high  priority  will  be  on  ensuring  that  modeling 
applications  for  the  Battlespace  Environments  Institute  are  completed  as  the  program 
draws  to  a  close.  Optimizations  for  GEOS-5  and  the  public  release  of  CCSM  will  be 
implemented.  Special  attention  will  be  paid  to  new  customers  such  as  the  NCAR  High 
Altitude  Observatory  TIMEGCM  model  developers,  who  have  started  using  ESMF 
regridding,  and  customers  who  have  been  waiting  for  ESMF  capabilities  to  become 
available,  such  as  the  developers  of  the  ROMS  ocean  model. 

6  Detailed  accomplishments  for  FY2009 

During  FY2009,  the  ESMF  team  made  significant  advances  in  software  development; 
training  and  support;  and  community  adoption.  The  following  sections  describe  metrics 
of  success  and  accomplishments  in  detail. 

6.1  Develop  software  for  grid  representation  and  regridding 

Measures  of  success: 

•  Delivery  of  ESMF  releases  capable  of  representing  multi-patch  logically 
rectangular  grids,  unstructured  grids,  and  observational  data  streams. 

•  Delivery  of  an  ESMF  release  that  generates  regridding  weights  internally  and  can 
perform  regridding  using  those  weights. 

Actual  performance,  outputs  and  outcomes: 

Work  in  FY09  on  unstructured  grids  has  included  continued  interaction  with  BEI  staff 
working  on  the  pWASH123-ADCIRC  coupling,  provision  of  necessary  interfaces  to 
complete  this  coupling,  including  C  interfaces,  improvements  to  the  unstructured  grid 
(Mesh  class)  documentation  and  more  Mesh  usage  examples,  and  various  bug  fixes. 
Bugs  have  tended  to  be  caused  by  non-portable  C++  features  used  in  the  Mesh 
implementation,  which  are  being  replaced  or  corrected.  Regridding  to/from  fields 
built  on  unstructured  grids  has  been  demonstrated  and  the  documentation  and 
examples  now  include  examples  of  this.  The  Mesh  code  and  regridding  software 
supports  meshes  with  elements  that  can  be  represented  as  triangles  or  quadrilaterals, 
in  2-D  or  3-D.  Improvements  to  the  Mesh  software  are  included  in  public  release 
ESMF  v4.0.0r,  which  was  released  in  October  2009. 

A  conservative  regridding  strategy  has  been  implemented  in  test  code.  It  performs  a 
local  conservative  correction  to  an  interpolation  algorithm  of  choice,  where  current 
options  in  ESMF  are  either  bilinear  or  higher  order.  An  advantage  of  the  algorithm  is 
that  derivatives  of  the  field  values  do  not  need  to  be  supplied  along  with  the  values. 
The  test  code  has  shown  good  preliminary  results  and  a  production  version  of  the 
algorithm  is  being  implemented  in  ESMF.  The  algorithm  was  developed  in 
collaboration  with  Amik  St.  Cyr,  a  mathematician  from  the  Institute  for  Mathematics 
in  the  Geosciences  (IMaGe)  at  NCAR.  The  conservative  algorithm  is  expected  to  be 
released  in  early  2010. 

Overall,  the  regrid  capability  in  ESMF  has  grown  increasingly  flexible,  with  options 
to  generate  weights  on-line  or  offline,  and  in  parallel  or  serial.  Currently  the  on-line 
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version  of  regridding  supports  all  the  grids  that  ESMF  does  (structured/unstructured), 
while  the  off-line  utility  only  supports  single-patch  logically  rectangular  grids.  The 
work  needed  to  create  interfaces  for  the  other  grid  types  in  the  off-line  version  is 
straightforward. 

The  items  that  were  identified  as  high  priority  last  year  were  fully  or  mostly 
completed  and  will  be  included  in  upcoming  releases  (C  interfaces  for  Mesh, 
regridding  for  Mesh,  observational  data  streams,  conservative  regridding).  However, 
a  shortcut  for  a  physical-space  representation  of  multi-patch  grids  was  not  completed; 
since  these  grids  can  be  represented  in  ESMF  using  lower-level  data  structures,  this 
tends  to  slip  down  the  priority  list.  The  implementation  of  exchange  grids  is  expected 
to  rely  upon  the  implementation  of  the  conservative  regridding  implementation,  and  it 
is  still  pending  as  well. 

6.2  Improve  usability 

Measures  of  success: 

•  1 00%  test  coverage  of  framework  methods. 

•  Delivery  of  a  release  that  further  improves  consistency  of  behavior  and  interfaces. 

•  Complete  sufficient  C  interfaces  to  meet  customer  needs. 

•  Maintain  or  reduce  the  number  of  open  bug  reports  and  feature  requests.. 

Actual  performance,  outputs,  and  outcomes: 

Test  coverage  is  at  100%  ,  meaning  that  all  public  methods  have  at  least  one  test  (e.g., 
this  is  not  a  100%  guarantee  of  system  correctness). 

The  4.0.0r  release  included  a  number  of  improvements  for  consistency  of  behavior 
and  interfaces,  including  the  introduction  of  standard  error  handling  to  Mesh  class 
methods,  and  standardized  handling  of  third-party  libraries. 

As  mentioned  in  the  previous  section,  C  interfaces  for  the  Mesh  class  were  delivered 
with  4.0.0r  in  order  to  complete  the  pWASH123-ADCIRC  coupling  for  BEI. 

Figure  1  (a)  shows  that  the  number  of  open  bug  reports  once  again  grew  after  a 
concerted  effort  reduced  the  number  by  40%  at  the  end  of  2008.  Intensive  testing 
before  the  4.0.0r  public  release  resulted  in  numerous  new  reports.  The  ESMF  team 
addressed  many  of  these  before  the  release,  and  others  will  be  fixed  in  a  forthcoming 
patch.  Many  of  the  bugs  that  remain  relate  to  gaps  or  deficiencies  in  documentation. 
The  plan  is  to  address  these  in  2010  in  the  ESMF  v5.0.0  release. 

Figure  1  (b)  shows  that  the  30%  decrease  in  feature  requests  at  the  end  of  2008  wasn’t 
sustained.  The  number  of  open  requests  increased  steadily  through  2009,  eventually 
exceeding  the  previous  peak.  In  general,  new  features  were  prioritized  lower  than 
bug  fixes  and  completion  of  the  capabilities  outlined  in  the  ESMF  Strategic  Plan,  and 
the  steady  stream  of  requests  simply  came  in  faster  than  they  could  be  processed. 
Exceptions  were  made  for  new  features  that  were  required  to  satisfy  application 
milestones. 
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New  vs.  Open  Bugs 


■  New  Bugs  ■  Open  Bugs 
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Figure  1.  Bug  and  feature 
request  metrics.  Following  a 
major  reduction  at  the  end  of 
FY09,  the  number  of  bugs  and 
feature  requests  inched  up 
again.  Chart  (a)  shows  the 
number  of  open  ESMF  bug 
reports,  chart  (b)  the  number  of 
ESMF  feature  requests. 


New  vs.  Open  Features 


■  Open  Features  ■  New  Features 
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6.3  Provide  ESMF  training  and  support 

Measures  of  success: 

•  Hold  two  training  classes  during  FY2009. 

•  User  comments,  surveys,  and/or  evaluations  show  that  ESMF  customers  are 
satisfied  with  support,  training,  and  documentation. 

•  Produce  a  journal  paper  on  ESMF. 

•  Develop  an  Outreach  Plan  to  promote  use  of  the  framework. 

Actual  performance,  outputs,  and  outcomes: 

A  “Getting  Started”  page  was  added  to  the  ESMF  website,  in  order  to  lay  out 
concepts  and  tasks  required  for  adoption  as  simply  as  possible  for  scientific  users.  A 
system  test  showing  a  complex  ensemble  example  was  added,  including  ensemble 
members  running  both  sequentially  and  concurrently  within  the  same  application. 
Additional  examples  were  added  to  the  Mesh  documentation,  and  improvements 
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continued  to  be  made  to  the  overall  documentation  based  on  support  requests  and 
other  customer  feedback.  Based  on  customer  comments,  they  are  increasingly 
satisfied  with  documentation  quality. 

The  ESMF  team  contributed  to  a  journal  article  entitled  “Integrated  Modeling  of  the 
Battlespace  Environment”  submitted  to  the  Computing  in  Science  and  Engineering 
journal  (lead  author  Tim  Campbell).  The  ESMF  also  participated  in  a  paper  entitled 
“A  Scientific  Workflow  Implementation  for  Earth  System  Science  Related 
Applications,”  submitted  to  the  Earth  Science  Informatics  journal,  that  described 
aspects  of  the  ESMF-enabled  Community  Climate  System  (lead  author  Turuncoglu). 
Several  book  sections  were  submitted  on  ESMF  for  a  Springer  volume  entitled  “Earth 
System  Modelling  Strategies  and  Software”  (lead  author  for  sections  DeLuca;  editors 
Budich  and  Redler),  and  also  to  a  volume  entitled  Handbook  of  Environmental  Fluid 
Dynamics  (lead  author  for  sections  DeLuca;  editor  Fernando). 

Tentative  arrangements  were  made  for  a  training  session  at  NCAR  in  late  summer 
2009;  however,  the  session  was  deferred  due  to  the  effort  and  arrangements  required 
in  moving  the  project  from  NCAR  to  NOAA.  The  Outreach  Plan  was  also  deferred. 

6.4  Develop  software  utilities  and  links  to  science  gateway  services 

Measure  of  success: 

•  Delivery  of  a  public  ESMF  release  that  includes  new  features  for  existing  utilities, 
prioritized  in  response  to  user  input. 

Actual  performance,  outputs,  and  outcomes: 

ESMF  4.0.0r  contained  an  initial  implementation  of  an  ESMF  I/O  class,  but  it  still 
requires  development  to  be  usable  by  production  applications. 

A  student  intern  from  the  University  of  Michigan,  Peter  Bosler,  collaborated  with 
NRL  to  produce  a  version  of  COAMPS  that  used  the  ESMF  Attribute  class  to 
instrument  the  model  with  metadata.  A  student  visitor  from  Istanbul  Technical 
University,  Ufuk  Turuncoglu,  did  the  same  for  CCSM,  and  embedded  this  version  in 
an  automated  workflow  that  configured  the  model,  ran  it,  and  documented  the  run. 
These  projects  are  building  towards  an  increased  level  of  automation  and  semantic 
interoperability  that  will  be  an  important  element  of  the  next  ESMF  development 
cycle.  Other  science-gateway  projects  are  briefly  outlined  in  Appendix  C. 

6.5  Framework  optimization  and  porting 

Measures  of  success: 

•  Minimal  performance  burden  for  working  code.  (Target  <5%  overhead  in 
component  overhead  and  regridding  methods.) 

Actual  performance,  outputs,  and  outcomes: 

Optimizations  were  also  performed  on  the  parallel  search  algorithm  used  in 
generating  interpolation  weights.  This  resulted  in  an  order  of  magnitude  reduction  in 
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the  time  required  for  the  cases  that  were  tested.  A  report  on  this  optimization  is  being 
prepared  and  will  be  posted  to  the  performance  page  on  the  ESMF  website  in  2010. 

In  2008,  initial  tests  of  the  sparse  matrix  multiply,  the  main  communication  kernel  for 
regridding,  were  performed  out  to  16K  processors.  A  variety  of  data  structures 
capable  of  representing  a  set  of  2D  fields  were  evaluated,  including  Arrays  and 
ArrayBundles.  The  report  for  this  has  been  completed  and  the  analysis  is  posted  at: 
http://www.esmf.ucar.edu/metrics/performance/timing  0901  asmm.pdf 

The  study  showed  low  absolute  times  for  the  communication  overall:  less  than  a  few 
msecs  for  all  configurations  at  more  than  128  processors. 

Users  and  internal  tests  indicate  that  continued  development  has  not  degraded 
performance  and  ESMF  still  represents  <5%  overhead  for  most  applications. 

6.6  Advance  community  adoption 

Means,  strategies,  and  actions: 

Assist  in  the  development  of  new,  coupled  ESMF  applications  and  the  transition  of 
these  applications  to  routine  use  and  operations. 

Measures  of  success: 

•  Number  of  ESMF  components  available.  A  goal  for  FY2009  was  completion  of 
an  ESMF  version  of  CCSM4. 

•  Number  of  ESMF  applications  in  routine  use  or  operations.  The  goal  for  FY2009 
include  having  at  least  two  more  ESMF  codes  transition  to  production  or 
operations. 

•  Number  of  programs  based  on  ESMF. 

Actual  performance,  outputs,  and  outcomes: 

The  ESMF  version  of  CCSM  was  completed,  There  are  now  about  90  ESMF 
components  that  have  been  implemented. 

•  The  WRF  code  has  had  optional  ESMF  bindings  in  their  distribution  since 

December,  2006. 

•  The  NASA  GEOS-5  atmospheric  general  circulation  model  based  on  ESMF  has 

been  in  production  since  2005. 

•  The  NCEP  GFS  with  ESMF  was  put  into  operations  in  August,  2006. 

•  The  coupled  HYCOM  and  CICE  model  using  ESMF  is  now  in  routine  use  at  the 

Naval  Research  Laboratory. 

•  The  CO  AMPS  and  NCOM  application  using  ESMF  components  and  coupling  has 

been  in  use  since  2007. 

MAP,  BEI  and  NUOPC  are  the  largest  programs  to  date  that  have  been  based  on 
ESMF. 
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7  Project  plan  evaluation  measures  for  FY2010 

The  sections  below  describe  key  metrics  and  goals  for  FY2009. 

7.1  Develop  software  for  grid  representation  and  regridding 

Measures  of  success: 

•  Delivery  of  an  ESMF  release  that  includes  the  ability  to  represent  and  regrid 
multi-tile  grids  and  exchange  grids. 

•  Delivery  of  an  ESMF  release  that  offers  integrated  conservative  regridding. 

•  Improved  integration  of  ESMF  with  external  regridding  capabilities,  including  an 
updated  version  of  SCRIP  and  libCF. 

7.2  Improve  usability 

Measures  of  success: 

•  Delivery  of  a  release  with  highly  consistent  interfaces  and  behavior,  ready  to  be 
frozen  for  backwards  compatibility. 

•  Maintain  or  reduce  the  number  of  open  bug  reports  and  feature  requests. 

7.3  Provide  ESMF  training  and  support 

Measures  of  success: 

•  Deliver  on-line  training  materials  to  accompany  the  ESMF  v5  release. 

•  User  comments,  surveys,  and/or  evaluations  show  that  ESMF  customers  are 
satisfied  with  support,  training,  and  documentation. 

•  Produce  one  or  more  journal  papers  on  ESMF. 

•  Develop  an  Outreach  Plan  for  2010-2015  to  promote  use  of  the  framework. 

7.4  Develop  software  utilities  and  science  gateway  services 

Measures  of  success: 

•  Delivery  of  a  public  ESMF  release  that  includes  new  features  for  existing  utilities 
and  an  I/O  library. 

•  Continued  development  of  ESMF  Attributes,  including  a  more  general  and  robust 
XML  write  capability. 

7.5  Framework  optimization  and  porting 

Measures  of  success: 

•  Minimal  performance  burden  for  working  code.  (Target  <5%  overhead  for  ESMF 
component  and  regridding  methods  vs.  native  packages.) 

•  Continued  demonstration  of  performance  viability  for  petascale  and  other 
emerging  computing  architectures. 

•  Requested  ports  delivered. 
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7.6  Advance  community  adoption 

Measures  of  success: 

•  Number  of  ESMF  components  available.  A  goal  for  2010  will  be  successful 
completion  of  all  the  BEI  coupled  applications. 

•  Number  of  ESMF  applications  in  routine  use  or  operations.  Goals  for  FY2010 
include  having  at  least  two  more  ESMF  codes  transition  to  production  or 
operations. 

•  Define  a  technical  and  programmatic  relationship  between  ESMF  and  other 
standard  community  interfaces,  particularly  OpenMI  in  the  hydrology  domain. 

8  Impact  of  the  ESMF  project 

Widespread  use  of  ESMF  represents  a  paradigm  shift  in  the  way  weather  and  climate 
models  are  constructed.  Through  ESMF  standard  interfaces,  community  governance,  and 
shared  tools,  large  centers  are  more  easily  able  to  collaborate  on  infrastructure  needs  and 
exchange  codes.  The  end  result  is  an  Earth  science  community  better  equipped  to 
explore  basic  research  issues  and  to  answer  questions  about  the  changing  environment. 

9  Sponsors 

Core  ESMF  development  is  sponsored  by  the  NSF,  NASA,  NOAA,  and  the  Department 
of  Defense.  In  addition,  a  variety  of  ESMF-based  application  adoption  projects  have 
been  sponsored  by  NASA,  NOAA,  the  U.S.  Geological  Survey/Department  of  the 
Interior,  and  other  agencies  and  institutions. 
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Appendix  A:  Table  of  Acronyms 


ADCIRC 

Advanced  Circulation  Model  for  Coastal  Ocean  Hydrodynamics 

AR5 

IPCC  5th  Assessment  Report 

BEI 

Battlespace  Environments  Institute 

CCSM 

Community  Climate  System  Model 

CICE 

Los  Alamos  sea  ice  model 

CMIP5 

5th  Coupled  Model  Intercomparison  Project 

COAMPS 

Coupled  Ocean/ Atmosphere  Mesoscale  Prediction  System 

CoG 

Commodity  Governance  project 

CSDMS 

Community  Surface  Dynamics  Modeling  System 

ESMF 

Earth  System  Modeling  Framework 

FMS 

Flexible  Modeling  System 

GEOS-5 

Goddard  Earth  Observing  System  Model 

GFDL 

Geophysical  Fluid  Dynamics  Laboratory 

GFS 

Global  Forecast  System 

GIP 

Global  Interoperability  Program 

GO-ESSP 

Global  Organization  of  Earth  System  Science  Portals 

HYCOM 

Hybrid  Coordinate  Ocean  Model 

IMaGe 

Institute  for  Mathematics  in  the  Geosciences 

IPCC 

Intergovernmental  Panel  on  Climate  Change 

MAP 

Modeling  Analysis  and  Prediction  program 

MAPL 

Modeling  Analysis  and  Prediction  program  Layer 

METAFOR 

common  Metadata  For  climate  modelling  digital  repositories 

MOM4 

Modular  Ocean  Model  4 

NASA 

National  Aeronautics  and  Space  Administration 

NCAR 

National  Center  for  Atmospheric  Research 

NEMS 

NOAA  Environmental  Modeling  System 

NESII 

NOAA  Environmental  Software  Infrastructure  and  Interoperability  group 

NOAA 

National  Oceanic  and  Atmospheric  Administration 

NRL 

Naval  Research  Laboratory 

NSF 

National  Science  Foundation 

NUOPC 

National  Unified  Operational  Prediction  Capability 

NWP 

Numerical  Weather  Prediction 

PCMDI 

Program  for  Climate  Model  Diagnosis  and  Intercomparison 

pWASH123 

parallel  WAterSHed  1-2-3D  model 

ROMS 

Regional  Ocean  Modeling  System 

SCRIP 

Spherical  Coordinate  Interpolation  and  Remapping  Package 

SWAN 

Simulating  WAves  Nearshore  model 

SWAT 

Soil  Water  Analysis  Testbed 

WRF 

Weather  Research  and  Forecast  model 
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Appendix  B:  Current  and  Pending  Support,  ESMF  and  Curator 


Program 

Sponsor 

Period  of  Performance 

Battlespace  Environments  Institute 
Infrastructure  Development 

DoD 

$  1  M/yr;  end  12/10 

Modeling  Analysis  and  Prediction 

Program  Infrastructure  Development  and 
Curator 

NASA 

$800K/yr;  end  12/10 

Science  Gateway  Development 

NSF 

TeraGrid 

$50K/yr;  8/09-8/11 

Curator  Portal  Workspaces  and 

Commodity  Governance  (CoG) 

NSF  CISE 

$300K/yr;  2/10-2/13 

NUOPC 

NOAA/DoD 

TBD/yr;  3/10-March  3/15 

Global  Interoperability  Program 

NOAA 

$lM/yr;  6/09-TBD 
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Appendix  C:  2009  Curator  Projects 


Project 

Sponsor(s) 

Partners 

Status 

1 .  Earth  System  Grid  gateway 
development  for  display  of 
model  metadata  for 

CMIP5/AR5  runs 

Why:  climate  data  distribution 

NOAA  and 
NASA 

Curator 

ESG, 

PCMDI,  EU 

METAFOR, 

NCAR, 

Georgia 

Institute  of 

Technology 

Delivered  in 
beta  ESG 
release  1/10 

2.  GUI  and  workflow  for  CCSM 
and  other  climate  model  runs 

Why:  easier  execution  of 
climate  models 

NSF  and  GIP 

NCAR, 
Purdue,  ESG 

Project  initiated 
in  1 1/09 

3.  CAM  coupling  with  Soil  Water 
Analysis  Testbed  (SWAT) 
model  to  demonstrate  ESMF 
interaction  with  OpenMI-based 
hydrology  codes 

Why:  local  application  of 
climate  model  information 

NASA 

Curator 

CUAHSI, 

use 

In  progress, 
estimated 
completion  9/10 

4.  COAMPS  metadata 

instrumentation  to  show  model 
self-documentation 

Why:  working  towards 

NUOPC  interoperability 
criteria 

NCAR 

SIParCS 

internship 

University  of 

Michigan, 

NRL 

Prototype 
completed  9/09, 
will  be  used  in 
NUOPC 

5.  Demonstration  of  ESMF  write 
of  CUAHSI  hydrology 
metadata,  shows  how  ESMF 
applications  can  produce  data 
for  a  hydrology  portal 

Why:  local  application  of 
climate  information 

NCAR 

SIParCS 

internship 

Louisiana 

State 

University, 

CUAHSI 

Prototype 
completed  9/09, 
may  be  used  in 
CAM-SWAT 
project  (3) 

6.  Self-documentation  of  CCSM 
in  a  Kepler  workflow 

Why:  easier  execution  of 
climate  models 

Istanbul 

Technical 

University 

Istanbul 

technical 

University, 

NCAR 

Prototype 
completed  9/09, 
will  be  used  in 
(2) 
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7.  Development  of  workspaces 
and  on-line  governance  for 
community  modeling,  esp. 
model  comparison  and 
validation  with  observations 

Why:  preparation  for  larger, 
more  diverse  communities  who 
need  Earth  system  models  and 
outputs 

NSF  CISE 

University  of 

Michigan, 

CSDMS, 

NCAR, 

University  of 

Colorado 

Project  about  to 
begin  in  2/10 

8.  Integration  of  ESMF  with  the 
Eclipse  Modeling  Framework 

Why:  easer  development  of 
coupled  models 

NSF 

fellowship 

Georgia 

Institute  of 

Technology 

(Curator 

minimally 

involved) 

Prototype 
completed,  may 
be  used  in 
compliance 
checking 
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